Papers with approximation error
The NLP Task Effectiveness of Long-Range Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks on long-range attention models have not been sufficient to develop efficient Transformers and their practical application on complex NLP tasks. |
| Approach: | They propose to benchmark 7 Transformer variants on 5 difficult NLP tasks and 7 datasets to examine their capacity for long-range attention. |
| Outcome: | The proposed models have advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error. |
Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a large computational cost for attention computation in large language models is a major obstacle . |
| Approach: | They propose a convolution-like structure for attention computation using convolution matrices . they then propose an efficient approximation method to approximate the attention matrix . |
| Outcome: | The proposed method achieves nearly linear time complexity in n1+o(1) time. |
A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning disentangled representations of textual data are difficult to implement and suffer from the degeneracy of other losses in multi-class scenarios. |
| Approach: | They propose a variational upper bound to the mutual information between an attribute and the latent code of an encoder that controls the approximation error. |
| Outcome: | The proposed method is superior on fair classification and on textual style transfer tasks. |
Efficient k-NN Search with Cross-Encoders using Adaptive Multi-Round CUR Decomposition (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ANNCUR uses a cross-encoder only to perform k-NN search, but the approximation of the distances is often detrimental to the retrieval of top-k items. |
| Approach: | They propose a method that minimizes approximation error for k-nearest neighbor searches . they propose to use a cross-encoder only to perform k NN search . |
| Outcome: | The proposed method reduces approximation error for top-k neighbors by up to 70% . iteratively performs k-NN search using the available anchors, then adds them to the next set . |